The Discrete Cosine Transform type two (DCT-II) is often used in signal processing, especially for lossy data compression, due to its strong “energy compaction” property. For image and video compression, the DCT without explicitly referring to a specific type implies the DCT-II. In the DCT-II representation, most of the signal information tends to be concentrated in a few low-frequency components. For signals based on certain limits of Markov processes, the DCT-II is approaching the Karhunen-Loève transform, which is optimal in decorrelating the underlying signals to obtain a most energy-compact representation.
The N-point DCT of the signal f[n] is defined as in (1).
                                                                        f                ^                            DCT                        ⁡                          [              k              ]                                =                                    λ              k                        ⁢                          2                              N                                      ⁢                                          ∑                                  n                  =                  0                                                  N                  -                  1                                            ⁢                                                f                  ⁡                                      [                    n                    ]                                                  ⁢                                  cos                  ⁡                                      [                                                                                            k                          ⁢                                                                                                          ⁢                          π                                                N                                            ⁢                                              (                                                  n                          +                                                      1                            2                                                                          )                                                              ]                                                                                      ,                                  ⁢                  k          =          0                ,        1        ,        2        ,        …        ⁢                                  ,                  N          -          1                ,                              λ            k                    =                      {                                                                                                      2                                              -                        0.5                                                              ,                                                                                        k                    =                    0                                                                                                                    1                    ,                                                                                        k                    ≠                    0                                                                                                          (        1        )            
In video coding, such as the state-of-the-art international video coding standard HEVC (high efficiency video coding), DCT is also used for its “energy compaction” property. FIG. 1 shows the block diagram of a video encoder utilizing transform coding. First, the original frame subtracts its prediction (Intra- or Inter-prediction) using an adder (110) and results in residual signal. In practice, the residual is often formed on a block by block basis. Second, the transform (120), such as DCT, is applied to the residual signal (i.e., prediction error) to obtain transform coefficients as output. Then the transform coefficients are quantized using a quantizer (130), which may cause signal distortion. Finally the quantized coefficients are entropy coded using an entropy coder (140) into a bitstream. In video coding standards, a picture may be partitioned into different units for different processing. For example, in high efficient video coding (HEVC), a picture is partitioned into coding tree units (CTUs) for applying a set of coding parameters. The CTU may be further partitioned into coding units (CUs). Each CU may be partitioned into one or more prediction units (PUs) to apply prediction (e.g., Inter- or Intra-prediction). The prediction residual for each CU is partitioned into transform units (TUs) to apply transform (e.g., DCT). In video coding, a picture may also be divided into multiple slices for packing compressed data or applying adaptive coding control.
In the current HEVC, the Discrete Sine Transform type seven (DST-VII) is used for 4×4 luma Intra-predicted residual, and DCT is used for all the other cases. It is desirable to improve the performance by incorporating other suited transforms to match characteristics of the underlying residual signals.